Image processing apparatus

ABSTRACT

In order to provide an image processing apparatus that can obtain images having a plurality of F-numbers simultaneously by one shooting operation, the image processing apparatus includes an image pickup device, each pixel of the image pickup device has a micro lens for condensing light and a photoelectric conversion region provided beneath the micro lens, and the photoelectric conversion region includes an upper electrode and a lower electrode sandwiching a photoelectric conversion film and forms at least first to third regions that are divided and arranged in a plane parallel to the image pickup plane by the upper electrode or the lower electrode. Additionally, the first region has a shape for forming a circle, and the second and third regions are arranged outside the first region. Additionally, the image processing apparatus includes a readout unit that reads out photoelectric conversion signals obtained from one image pickup operation from each of the first to third regions.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus that isappropriate for obtaining images having different F-numbers.

Description of the Related Art

Recently, there has been proposed an image pickup apparatus that isprovided with image pickup device that can obtain an image for imagerecording or display and can perform phase difference focus detection onan image pickup plane, in other words, image pickup device having whatis referred to as an “image pickup plane phase difference AF function”.Further, there has been proposed a camera that can acquire a pluralityof pieces of light ray space information. Such a camera is referred toas, for example, a “light field camera”. The light field camera canprovide a function such as changing a focus position after shooting andobtaining images having different F-numbers (aperture values) byobtaining light fluxes that have passed through different pupil regionsand reconstructing the image.

Additionally, in many cases, a CCD or a CMOS image pickup device or thelike is used in an image pickup apparatus such as a digital still cameraand a digital video camera. The image pickup device photoelectricallyconverts light that is incident to a photodiode for each pixel formed ona semiconductor substrate and reads out a signal amount for each pixel.Recently, in order to improve a light receiving sensitivity, a stackedimage pickup device in which an organic photoelectric conversion filmformed between a lower electrode and an upper electrode and a colorfilter are stacked has been disclosed. In an image pickup device havingan organic photoelectric conversion film, setting a light receivingregion of an image pickup device for receiving light of different pupilregions is relatively easier than in a conventional CCD or CMOS imagepickup device. Specifically, there is the characteristic that the degreeof freedom for pattern formation corresponding to the pupil region ofthe image pickup device is high because the light region can be set by alower electrode pattern made of a metal material.

In an image pickup apparatus disclosed in Japanese Patent No. 05917158,a plurality of micro lens arrays are provided and a plurality of imagepickup elements are provided beneath micro lenses so as to configure theabove light field camera. A method for obtaining an image correspondingto a predetermined F-number of the image pickup device by changingreadout of the image pickup device is also disclosed. Additionally,Japanese Patent Application Laid-open No. 2013-145292 discloses an imagepickup device having an image pickup plane phase difference AF functionrealized by structures of lower electrodes, in an image pickup devicehaving an organic photoelectric conversion film.

However, in the image pickup apparatuses disclosed in Japanese PatentNo. 05917158 and Japanese Patent Application Laid-open No. 2013-145292,there is a drawback to be described below when acquiring images havingdifferent F-numbers. In the image pickup apparatuses disclosed inJapanese Patent No. 05917158 and Japanese Patent Application Laid-openNo. 2013-145292, a divided shape of the image pickup element isrectangular, and thus, when an image formed by reading out of the imagepickup element (for example, an image corresponding to a large F-number)is formed, a shape of blur becomes rectangular corresponding to theshape of the image pickup element. A mechanical diaphragm of a typicallens is changed so as to keep a round shape of aperture in accordancewith the F-number. In contrast, a rectangular blur is not preferable.

SUMMARY OF THE INVENTION

An object of the present invention is to provide an image processingapparatus that can easily obtain a plurality of images corresponding todifferent F-numbers and can optimize a blur shape.

In order to solve the above problems, an image processing apparatusaccording to one aspect of the present invention comprises: an imagepickup device configured to have a plurality of pixels arranged along animage pickup plane, the pixel having a micro lens for condensing lightfrom outside into the pixel, and a photoelectric conversion regionprovided beneath the micro lens and for generating a photoelectricconversion signal, the photoelectric conversion region including anupper electrode and a lower electrode sandwiching a photoelectricconversion film, the upper electrode or the lower electrode beingdivided into a plurality of portions, and forming at least a firstregion, a second region, and a third region that are divided andarranged in a plane parallel to the image pickup plane by the dividedupper electrodes or the lower electrodes, the first region having ashape for forming a circle, the second region and the third region beingarranged outside the first region; and a readout unit configured to readout the photoelectric conversion signals obtained by one image pickupoperation to serve as respectively first, second, and third signals fromthe first, second, and third regions.

Further features of the present invention will become apparent from thefollowing description of experimental artifacts with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image processing apparatus according toEmbodiment 1.

FIG. 2 illustrates an example of pixel arrangement of image pickupdevice according to Embodiment 1.

FIG. 3 illustrates a sectional view of pixels of the image pickup deviceaccording to Embodiment 1.

FIG. 4 is a schematic diagram illustrating the correspondence betweenthe image pickup device and the pupil division according to Embodiment1.

FIGS. 5A and 5B illustrate an example of a pattern of a pupil region ofan imaging optical system according to Embodiment 1.

FIG. 6 is a schematic diagram illustrating the relation between anamount of image shift and an amount of defocusing according toEmbodiment 1.

FIG. 7 illustrates the relation between the pupil region and theF-number according to Embodiment 1.

FIG. 8 illustrates an example of pixel arrangement according toEmbodiment 1.

FIG. 9 is a sectional view of the pixels in the pixel arrangement ofFIG. 8.

FIGS. 10A-10D illustrate another example of a pattern of the pupilregion.

FIGS. 11A-11D illustrate an example in which the pupil region is dividedinto many portions according to Embodiment 2.

FIG. 12 is a sectional view of the pixels in the pixel arrangementaccording to Embodiment 2.

FIGS. 13A-13C illustrate a modification of the pupil division accordingto Embodiment 2.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention will be describedbelow in detail with reference to the drawings based on each embodiment.

Embodiment 1

FIG. 1 is a block diagram of a camera body 100 such as alens-interchangeable digital camera and a shooting lens unit 500 toserve as an example of an image processing apparatus that embodies thepresent invention. The shooting lens unit 500 is configured to beattachable to and detachable from the camera body 100 and light fluxestransmitted through each lens group in the shooting lens unit 500 areguided to the image pickup device 101 for receiving an object image. Inthe image pickup device 101, a plurality of pixels thatphotoelectrically convert an object image into an electric signal arearranged in a matrix.

Various types of correction processing for acquiring an image signal anda focus detection signal, signal processing for converting the acquiredimage signal into a live-view image, an image for recording image, or animage for display on an EVF and the like are performed on the pixelinformation that has been converted into the electric signal. Theseprocessing are performed by a camera CPU 104. The camera CPU 104performs various signal processing based on a computer program stored ina memory (not illustrated) and executes various operations for theoverall image processing apparatus. Although in the present embodimentthese processes and the like are performed by the camera CPU 104, a partof these processes may be performed by a dedicated circuit.

Reference numeral 105 is an operating member, which includes variousmembers for setting a shooting mode and a shooting condition (forexample, F-number, ISO, and exposure time) of the camera. Referencenumeral 106 is a storage medium that is configured by a flash memory andthe like, and functions as a storage means for recording shot stillimages and moving images. The storage medium 106 also has a function forindividually storing signals for a plurality of pupil regions that havebeen read out from the image pickup device 101 or for storing signalsafter partially adding the signals. Reference numeral 107 is aviewfinder display, which is configured by a display 109 serving as asmall and high-definition display means, such as an organic EL displayor a liquid crystal display, and an eyepiece lens 108. As an externaldisplay device 110, an organic EL display or a liquid crystal displayhaving a screen size that is suitable for naked eye vision is used.Various types of information, such as a setting state of the camera body100, a live view image, and a shot image, are displayed on theviewfinder display 107 and the external display device 110.

A focal plane shutter 111 is disposed in front of the image pickupdevice 101. A shutter driving unit 112 includes, for example, a motor,and controls an exposure time of the image pickup device 101 whencapturing a still image by driving and controlling shutter blades. Acamera-side communication terminal 113 is provided on a camera mountunit for mounting a shooting lens unit. The camera-side communicationterminal 113 transmits and receives various types of information betweenthe camera CPU 104 and a lens CPU 507, which will be described below,together with a lens-side communication terminal 508 provided on thelens mount unit.

The shooting lens unit 500 is attachable to and detachable from thecamera body 100 and, in the present embodiment, it has a zoom lens bywhich a focal length is variable. Any type of lens can be used. Thelight flux from the object passes through a first lens group 501, asecond lens group 502, and a third lens group 503, and forms an objectimage on an image pickup plane of the image pickup device 101 in thecamera body 100. The second lens group 502 functions as a variator thatmoves back and forth in the optical axis direction to changemagnification. The third lens group 503 functions as a focus lens thatmoves back and forth in the optical axis direction to perform focusadjustment.

The third lens group 503 is driven by a focus driving unit 504 thatincludes a stepping motor. A diaphragm 505 is used for adjusting anamount of light incident to the image pickup device 101 via the shootinglens unit, and is configured by a plurality of diaphragm blades. Adiaphragm driving unit 506 drives the diaphragm blades to drive thediaphragm until a predetermined F-number is achieved. The lens CPU 507communicates with the camera CPU 104 via the lens-side communicationterminal 508 and the camera-side communication terminal 113 to transmitand receive various types of information and controls the focus drivingunit 504 and the diaphragm driving unit 506 based on a command from thecamera CPU 104.

Although a zoom range and an open F-number of the shooting lens unit 500are designed in accordance with a shooting intention, in the presentembodiment, the open F-number is set to a constant value regardless of azooming state and a focusing state. In contrast, a distance between anexit pupil and an image pickup plane, in other words, what is referredto as an “exit pupil distance”, changes in accordance with the zoomingstate and the focusing state.

FIG. 2 is a plan view of a part of a plurality of pixels 200 in thevicinity of the center of the image pickup plane of the image pickupdevice 101 (in the vicinity of the optical axis of the shooting lensthat is in the vicinity of the image height 0) as viewed from theimaging lens unit side. In FIG. 2, the pixels 200 included in the imagepickup device 101 are respectively square-shaped pixels having the sizeof 4 μm in both the horizontal direction (x) and the vertical direction(y) on the image pickup plane, and the structure of these pixels issubstantially the same. These pixels are, for example, arranged in amatrix of 6,000 pixels in the horizontal direction and 4,000 pixels inthe vertical direction. The size of the image pickup region can beobtained by multiplying the size of the pixels, in other words, thepixel pitch, by the number of pixels, and in this case, it is 24 mm inthe horizontal direction and 16 mm in the vertical direction. RGB Colorfilters are arranged in each pixel, where what is referred to as “Bayerarrangement” is formed as a whole.

FIG. 3 is a sectional view in the x-direction passing through the centerof one pixel of the pixel group. The pixel 200 has a micro lens 201, acolor filter 202, an upper electrode 203, an organic photoelectricconversion film 204, and transparent lower electrodes 205A to 205D. Themicro lens 201 condenses light, which is imaged on the image pickupdevice 101 after passing through the shooting lens unit 500, onto aninside of the pixel 200. The color filter 202 is provided beneath themicro lens 201 and is configured to transmit only light having aspecific wavelength of, for example, R (red system), B (blue system), orG (green system) for each pixel 200.

The upper electrode 203 (first electrode) is provided beneath the colorfilter 202 and on the organic photoelectric conversion film 204. Theorganic photoelectric conversion film 204 is provided beneath the upperelectrode 203 and generates electric charges that serve as photoelectricconversion signals by photoelectric conversion of light. The lowerelectrodes 205A to 205D are second electrodes arranged separately in thex direction and are provided beneath the organic photoelectricconversion film 204 so as to face the upper electrode 203. The lowerelectrode is divided and arranged in a plane parallel to the imagepickup plane. The color filter 202 is provided between the micro lens201 and the upper electrode 203. The upper electrode 203 is providedbetween the color filter 202 and the organic photoelectric conversionfilm 204. The organic photoelectric conversion film 204 is providedbetween the upper electrode 203 and the lower electrodes 205A to 205D.

In each pixel 200, only light having a specific wavelength (R, G, or B)condensed by the micro lens 201 and transmitted through the color filter202 is photoelectrically converted by the organic photoelectricconversion film 204 to generate signal electric charges. Subsequently, abias voltage is applied between the upper electrode 203 and the lowerelectrodes 205A to 205D so as to form an electric field in the organicphotoelectric conversion film 204, whereby a signal electric charge thatis transferred to the lower electrode 205 can be read out to theoutside. That is, the photoelectric conversion signal that has beengenerated in the organic photoelectric conversion film is read out foreach region corresponding to the shape of the lower electrodes 205A to205D. That is to say, a photoelectric conversion region in accordancewith the shape of the lower electrode is formed. In the presentembodiment, the “photoelectric conversion region” denotes a regionformed by the divided lower electrode together with the upper electrodeand the organic photoelectric conversion film. Note that the upperelectrode may be divided into a plurality of electrodes. Therefore, thephotoelectric conversion region may be divided into a plurality ofregions by the divided upper electrodes.

The upper electrode 203 is transparent and allows light to enter intothe organic photoelectric conversion film 204 from the micro lens 201.The material of the transparent upper electrode 203 is preferably atransparent conductive oxide such as ITO (indium tin oxide).Additionally, the upper electrode 203 may be divided for each pixel 200or may be shared for all the pixels 200.

The lower electrodes 205A to 205D are metal electrodes having thefunction for reflecting light. Examples of the material of the lowerelectrodes 205A to 205D include Al and Ti. The lower electrodes 205A to205D reflect light incident from the organic photoelectric conversionfilm 204 to the organic photoelectric conversion film 204. The organicphotoelectric conversion film 204 is made of a photoelectric conversionmaterial that absorbs light and generates electric charges in accordancewith an amount of absorbed light. The organic photoelectric conversionfilm 204 may have a single-layer structure or a multi-layer structure.An insulating film 206 and a metal wiring 207 are provided beneath thelower electrodes 205A to 205D. The metal wiring 207 is an electricwiring portion for reading signal electric charges or switching signals,and the lower electrodes 205A to 205D are respectively connected to thesignal reading units 209A to 209D on the Si substrate. Note that thesignal electric charges that have been generated in the organicphotoelectric conversion film 204 and transferred to the lower electrode205 are accumulated in the corresponding signal reading units 209A to209D via the metal wiring 207. Specifically, a still image and a movingimage are generated by signals acquired based on the signal electriccharges accumulated in the signal reading units 209A to 209D during apredetermined exposure period of time.

The insulating film 206 is formed between a plurality of metal wirings207 and between a plurality of lower electrodes 205A to 205D. Thus, theinsulating film 206 provides electrical insulation between the metalwirings 207 and between the lower electrodes 205A to 205D. An impurityregion corresponding to the pixels 200 is formed in a semiconductorsubstrate 208 and holds the signal electric charges that have beengenerated by the organic photoelectric conversion film 204. Further, acircuit for reading out signal electric charges (not illustrated) isformed on the semiconductor substrate 208.

Next, FIG. 4 illustrates the correspondence relation between the imagepickup device and the pupil division in the present embodiment. A line604 indicates the position of the object, and an object image is formedon an image pickup plane 606 of the image pickup device through theimage pickup optical system located at a position 605. Note that in thiscontext, the imaging optical system represents a lens group included inthe shooting lens unit as one lens. The photoelectric conversion filmportions each corresponding to each of the lower electrodes 205A to 205Ddivided into four portions in the x direction for each pixel of theimage pickup device receive a light flux that has passed through pupilregions 607A to 607D shown by a dotted line.

In the present embodiment, a parallax image corresponding to a specificpupil region in the pupil regions 607A to 607D of an image-formationoptical system can be obtained by selectively selecting a signal of aspecific photoelectric conversion portion from among the photoelectricconversion film portions corresponding to the lower electrodes 205A to205D for each pixel. For example, a parallax image having a resolutionfor the effective pixel number corresponding to the pupil region 607A ofthe image-formation optical system can be acquired by selecting a signalof the photoelectric conversion film portion corresponding to the lowerelectrode 205A for each pixel.

Additionally, it is possible to generate a captured image having arelatively high sensitivity with a resolution for the effective pixelnumber by adding all or part of the signals of the photoelectricconversion portions corresponding to the photoelectric conversion filmportions corresponding to the lower electrodes 205A to 205D for eachpixel.

FIG. 5A is a schematic diagram illustrating the pupil regions 607A to607D of the imaging optical system on the xy plane. A pupil region 6073shown by a dotted line is a circular pupil region determined(restricted) by the diaphragm within the overall pupil region configuredby the pupil regions 607A to 607D in a manner similar to those in FIG.4. Each of the pupil regions 607A to 607D represents a pupil regiondetermined by the shapes of the lower electrodes 205A to 205D. That is,the pupil region shape (pupil pattern) shown in FIG. 5B in a broken-downmanner is determined in accordance with the shape of the lowerelectrodes 205A to 205D of the pixel 200 shown in FIG. 3. Note thatvignetting occurs due to the lens diaphragm in a portion outside theperiphery of the pupil regions 607A and 607D. In the present embodiment,as shown in FIG. 5B, a pupil pattern divided into 4 parts and consistingof pupil regions 607B and 607C (hatched portion) and 607A and 607D (dotportion) each having a substantially semicircular shape is illustrated.The 607A and 607D have a shape obtained by respectively removing 607Band 607C from the rectangles 6071 and 6072.

Additionally, the lower electrodes 205A to 205D respectively have shapesthat are similar to the shapes of the corresponding pupil regions, andas described above, photoelectric conversion regions in which the lowerelectrode is divided together with the upper electrode and the organicphotoelectric conversion film 204 are formed. That is, the divided pupilregions shown in FIG. 5A may be considered to have a shape substantiallysimilar to the shape of the divided lower electrodes 205A to 205D (ashape of the divided photoelectric conversion region). Additionally, inthe description below, for example, all of the divided pupil regions tobe described with reference to FIG. 7, FIGS. 10A-10D, FIGS. 11A-11D, andFIGS. 13A-13C may be considered to have a shape similar to the shape ofthe divided lower electrode (the shape of the divided photoelectricconversion region). As described above, since the pupil pattern can bedetermined by the lower electrode that is a metal material, the imagepickup device according to the present embodiment that uses aphotoelectric conversion film has higher degree of freedom whendesigning the pupil pattern than a conventional CCD or CMOS image pickupdevice. Additionally, there is the characteristic that the pupil patterncan be easily formed even in a complicated shape such as an arc.

As described above, in the present embodiment, semicircular lowerelectrodes 205B and 205C each having a shape that is similar to thepupil regions 607B and 607C are provided in order to constitute thesemicircular pupil regions 607B and 607C. Lower electrodes 205A and 205Dthat are respectively shaped similarly to the pupil regions 607A and607D are provided outside the lower electrodes 205B and 205C. Aphotoelectric conversion region (first region) formed by thesemicircular lower electrodes 205B and 205C has a shape for forming acircle by the combination of both. In the example of FIG. 5A, althoughthe lower electrodes 205B and 205C are divided, the photoelectricconversion region (first region) for forming a circle may not be dividedas shown in FIG. 10D and FIG. 13C, which will be described below, or maybe further divided into smaller parts as shown in FIG. 11A. Note that inthe above description, the circle does not have to be perfectlycircular, and, for example, it may be slightly oval.

Additionally, the lower electrodes 205A and 205D for forming dividedphotoelectric conversion regions (second region, third region) aredisposed outside the photoelectric conversion region (first region) forforming a circle as described above. Note that in the example shown inFIG. 5A, although one each of the lower electrodes 205A and 205D forforming the divided photoelectric conversion regions (second region,third region) are provided, they may be further divided into smallerportions as shown in FIG. 11A and FIGS. 13A-13C.

Additionally, in the present embodiment, as shown in FIG. 4, as thepixel is located farther from the center of the image pickup device, theposition of the micro lens is shifted to the optical axis side of theoptical system with respect to the pixel. This is in order to correspondto an inclination of the direction of the principal ray because, in aportion far from the center of the image pickup device, the direction ofthe principal ray from the image-forming optical system is moreinclined.

Next, FIG. 6 illustrates a schematic relation between an amount of imageshift between parallax images in each pixel and an amount of defocusing.Each of the micro lenses 201 of the image pickup device 101 in thepresent embodiment are arranged on the image pickup plane 606, and lightfrom the pupil region 607A to the pupil region 607D of the image formingoptical system enters in a manner similar to that in FIG. 4.

The defocus amount d is defined to have a magnitude |d| of the distancebetween the image forming position of the object and the imaging plane.Also, a negative sign (d<0) is given when in a front-focused state inwhich the image forming position of the object is on the object siderelative to the imaging plane, and a positive sign (d>0) is given whenin a rear-focused state in which the image forming position of theobject is on the side opposite to the object relative to the imagingplane. When in an in-focus state in which the image forming position ofthe object is on the imaging plane (in-focus position), d=0. In FIG. 6,an object 701 is an example of the in-focus state (d=0), and an object702 is an example of the front-focused state (d<0). The front-focusedstate (d<0) and the rear-focused state (d>0) will be collectivelyreferred to as a “defocused state (|d|>0)”.

In the front-focused state (d<0), each light flux that has passedthrough the pupil regions 607A to 607D, from among the light fluxes fromthe object 702, is condensed once and expanded to a width F centered onthe centroid position G_A to G_D of each light flux to form an imageblurred on the image pickup plane 606 (blur image). The blur image Γ_Ato Γ_D of each light flux is received by the photoelectric conversionfilm portion corresponding to the lower electrode 205, and thereby aparallax image is generated.

With the increase in the size “|d|” of the defocusing amount d, the blurimage Γ_A to Γ_D of the object image is generally widened in a directionparallel to the image pickup plane 606 in a proportional manner.Similarly, with the increase in the size “|d|” of the defocusing amountd, the size of the image shift amount p_AD (=G_D−G_A) of the objectimage between parallax images “|p|” generally increases in aproportional manner. The front-focused state being applied to therear-focused state (d>0), in which the direction of image shift of theobject image between parallax images is opposite to the front focusstate, is similar. In the in-focus state (d=0), the centroid position ofthe object image between the parallax images coincides (p=0), and imageshift does not occur.

At this time, the image shift amount p may be calculated as G_D−G_A ormay be calculated as G_C−G_B. Alternatively, the image shift amount maybe calculated as (G_C+G_D)−(G_B+G_A). The above three calculationmethods may be selectively switched in accordance with the defocusingamount (size of |d|) or addition may be performed after changing theweighting of the above three calculation results. In focus detection,the defocusing amount d is calculated by multiplying the image shiftamount p between parallax images by a known conversion coefficient K.Additionally, a lens driving amount is determined based on the relationbetween the defocusing amount and the third lens group 503 serving as afocus lens, and the image pickup plane phase difference focus adjustmentis performed by operating the focus driving unit 504. The aboveoperation is executed by the focus detecting means in the presentembodiment.

As described above, the amount of image shift between two or more (aplurality of) parallax images acquired by using the photoelectricconversion film portion corresponding to the lower electrode 205increases with the increase in the amount of defocusing. In the presentembodiment, the amount of image shift between parallax images iscalculated by using a correlation calculation by using signals from thephotoelectric conversion unit of the image pickup device. Accordingly,it is possible to perform focus detection by using a focus detectionsignal of the image pickup plane phase difference detection method.Thus, phase difference focus detection becomes possible by forming apupil pattern having a parallax.

Next, with reference to FIG. 7, the relation between captured imageshaving a plurality of F-numbers that can be acquired in the presentembodiment and a parallax image for focus detection will be described.FIG. 7 illustrates the relation with the F-number shown by a dotted linein the pupil region pattern shown in FIG. 5A. The pupil region 6073 is apupil region determined by an open diaphragm of the shooting lens unit500. For example, if the open F-number of the shooting lens unit is F2.0, a pupil region having F 2.8 (first-step aperture), and a pupilregion having F 4.0 (second-step aperture) are respectively shown bydotted lines in FIG. 7. Upon obtaining an addition output of the pupilregion 607A+the pupil region 607B+the pupil region 607C+the pupil region607D in an open aperture state, a shooting image for image recording orimage display having the open F-number F 2.0 can be acquired.

Note that, in order to acquire such an addition output, an image pickup(shooting) operation is performed once by using the image pickup deviceto form photoelectric conversion signals in the image pickup device.Subsequently, the photoelectric conversion signals (first to thirdsignals) from the photoelectric conversion regions (first region tothird region) corresponding to each of the pupil regions are read out.This readout is performed under the control of the camera CPU 104. Atthis time, the camera CPU 104 functions as a readout unit. AD conversionis performed on the signals that have been read out by an AD converter(not illustrated), signal processing is appropriately performed to theAD-converted signals, and then the signals are temporarily stored in thestorage medium 106. Then, addition processing is performed on thetemporarily stored signals corresponding to each pupil region by anarbitrary combination. Note that a part of the signals corresponding toeach pupil region may be added inside the image pickup device or outsidethe image pickup device in advance before being stored in the storagemedium 106. Alternatively, all the signals corresponding to each pupilregion may be separately stored in the storage medium 106.

FIG. 9 illustrates a cross-sectional structure of the pixels in thex-direction passing through the pixel center when the pixels 200, 300,and 400 are arranged in line as shown in FIG. 8. Additionally, thesignals from the signal reading units 209A to 209D of the pixel 200 arerespectively defined as 209AS to 209DS. Additionally, the signals fromthe signal readout units 309A to 309D of the pixel 300 are respectivelydefined as 309AS to 309DS. Additionally, the signals from the signalreading units 409A to 409D of the pixel 400 are respectively defined as409AS to 409DS.

For example, in the first mode, the signals 200AS+209BS+209CS+209DS areread from the pixel 200, the signals 300AS+309BS+309CS+309DS are readfrom the pixel 300, the signals 400AS+409BS+409CS+409DS are read fromthe pixel 400 . . . , and such reading processing is continued until thereadout for all pixels is completed (in the present embodiment, forexample, it is assumed that 4,000×6,000 pixels are arranged), andthereby, a shooting for image recording or image display correspondingto the open F-number (F2.0) is obtained. Specifically, in the firstmode, an image signal having a relatively high sensitivity can be formedby adding the first to third signals corresponding to the first to thirdregions.

In contrast, for example, in the second mode, the signals 209BS+209CSare read from the pixel 200, the signals 309BS+309CS are read from thepixel 300, the signals 409BS+409CS are read from the pixel 400 . . . ,and such reading processing is continued until the readout for allpixels is completed (in the present embodiment, for example, it isassumed that 4,000×6,000 pixels are arranged.), thereby to obtain ashooting image for image recording or image display corresponding to theopen F-number (F4.0). Specifically, instead of using the second andthird signals among the first to third signals that correspond to thefirst to third regions, in the second mode, an image signal having arelatively low sensitivity is formed by forming a second image signal byusing the first signal corresponding to the first region.

Note that the operation for selecting the first or second mode in theformation of such an image signal is performed based on, for example,object brightness information (photometric information). That is, forexample, if it is determined that the object is relatively dark based onthe photometric information, the first mode is selected, and incontrast, if it is determined that the object is relatively bright, thesecond mode is selected. Note that, in the present embodiment, theselection of the first or second mode is executed by the camera CPU 104,and at this time, the camera CPU 104 functions as an image processingmeans.

Note that the image signal forming operation in the first and secondmode may be executed by reading out the first to third signals from theimage pickup device, performing AD-conversion, storing the AD-convertedsignals in the storage medium, and appropriately combining the first tothird signals from the storage medium. Moreover, in the presentembodiment, the lower electrodes 205B, 205C and the like are each formedto be substantially semicircular such that the pupil shape also has asubstantially circular shape in a manner similar to the shape in whichthe diaphragm 505 is narrowed, to acquire a substantially circular blurshape, and as a result, a shooting image having a preferable blur shapecan be obtained. According to the present embodiment, the lowerelectrodes 205B and 205C are formed to be substantially circularcorresponding to the pupil regions 607B and 607C in FIG. 5A and FIG. 7,and as a result, blurring can be optimized.

Conventionally, there has been a necessity to shoot a plurality ofimages by operating the diaphragm driving unit 506 in the shooting lensunit 500 each time, in the acquisition of images having differentF-numbers. However, in the present embodiment, it is possible to acquireimages having different F-numbers by one shooting operation simply bychanging the combination of the addition signals from the signal readoutunit as described above, while changing the diaphragm by the diaphragmdriving unit 506, instead of shooting a plurality of times.Alternatively, images having different F-numbers obtained by oneshooting operation may be acquired by temporarily storing the signalsthat have temporarily been read out from the signal readout unit in thestorage unit, and then changing the combination of the addition signals.

Additionally, in focus detection, for example, control is performed asbelow. That is, signals 209AS and 209DS are respectively read out toserve as, for example, a right pixel signal and a left pixel signal fromthe pixel 200. The signals 309AS and 309DS are respectively read out toserve as, for example, a right pixel signal and a left pixel signal fromthe pixel 300. Then, signals 409AS and 409DS are respectively read outto serve as, for example, a right pixel signal and a left pixel signalfrom the pixel 400. Such an operation is continued in order to obtain,for example, a right image signal and a left image signal from pixels ofone row. Subsequently, a focus detection signal is generated bydetecting a phase difference between the right image signal and the leftimage signal, and an object distance is calculated. The same applies tothe other rows. In the present embodiment, a focus detection signal isgenerated based on the third signal from the third region and the fourthsignal from the fourth region within the photoelectric conversionregion. The generation of such a focus detection signal is controlled bythe camera CPU 104. At this time, the camera CPU 104 functions as afocus detection means.

Further, as shown in FIG. 10A, the pattern shape of the lower electrodecan also be configured so as to have a shape in which the pupil patternis rotated by 90 degrees with respect to FIG. 7. As shown in FIG. 10Band FIG. 10C, the pattern shape of the lower electrode can also beconfigured so as to have a shape in which the pupil pattern is rotatedby +45 degrees or −45 degrees with respect to FIG. 7. In FIG. 7, FIG.10A, FIG. 10B, and FIG. 10C, each parallax direction is different and adetection capability in the angular direction of the object isdifferent. Accordingly, if the pixels of the pattern of the lowerelectrode shown in FIG. 7, FIG. OA, FIG. 10B, and FIG. 10C are arrangedin a periodically mixed manner in the xy plane of the image pickupdevice 101 shown in FIG. 8, for example, the focus detection capabilitycan be improved.

Additionally, as shown in FIG. 10D, the pupil region 607B at the centermay be an undivided shape. In this case, the phase difference focusdetection may be performed in the pupil regions 607A and 607D.

In this way, in the pixel having the lower electrode corresponding tothe pupil pattern, the readout region of the pixel signal is changed,and the addition method of the signals that have been read from eachlower electrode is changed. Hence, images having a plurality ofF-numbers can be simultaneously acquired by one shooting, and an imagehaving a desired F-number can be acquired from among the images. Thesimultaneous acquisition of images having different F-numbers may be setby a user during the shooting operation, and may be processed in thecamera (image pickup apparatus). Alternatively, after shooting, anexternal device such as a personal computer and an electronic terminalother than the camera (image pickup apparatus) may be used to performpost-processing, and an image having a desired F-number may beselectively formed.

Specifically, the image pickup device and the readout means for readingout signals from the image pickup device are built in the camera (imagepickup apparatus). In contrast, the image signal processing means thatperforms addition processing to the readout signals to form an imagesignal for image recording or display may be built in the camera (imagepickup apparatus) or may be built in an external device other than thecamera (image pickup apparatus). Note that, in order to acquire imageshaving different F-numbers during post-processing, while shooting,signals corresponding to the lower electrodes, for example, 209AS to209DS, 309AS to 309DS, 409AS to 409DS . . . are read out independentlyand recorded temporarily. Subsequently, during post-processing, thesesignals are added or subtracted by a predetermined combination, so thatimages having different F-numbers can be acquired.

In this way, a plurality of pupil patterns having a pupil regioncorresponding to a shooting diaphragm and a pupil region having aparallax for focus detection are formed so that a shooting imagecorresponding to a plurality of F-numbers can be obtained by oneshooting simultaneously with phase difference focus detection. Moreover,since the pupil region can be formed to be, for example, substantiallycircular, a natural blur shape can be obtained.

Embodiment 2

Next, in Embodiment 2, an example in which the number of divisions ofthe pupil region is higher than that of Embodiment 1 will be describedwith reference to FIGS. 11A to 11D. FIG. 11A illustrates a pattern of alower electrode of Embodiment 2, and FIG. 11B schematically illustrates12 pupil regions that have been divided. As described above, anarbitrary pupil pattern can be formed by using the pattern of the lowerelectrode that is a metal material. Thus, in the image pickup devicehaving the photoelectric conversion film according to the presentembodiment, the degree of freedom for the pupil pattern shape is high.Even the complicated pupil pattern shape shown in FIG. 11A in Embodiment2 can be formed easier than a conventional CCD or CMOS image pickupdevice.

In the examples of FIG. 11A and FIG. 11B, the pupil region is dividedinto 12 portions from 707A to 707L, and as shown in FIG. 11B, the pupilregions 707C, 707F, 707, and 707L each has a circular shape divided intofour portions, and the pupil regions 707B, 707E, 707H, and 707K each hasa substantially annular shape divided into four portions. Additionally,the pupil regions 707A to 707L are regions having the shapes shown inFIG. 11B. As described above, these pupil shapes are similar to theshape of the lower electrodes of each pixel. Specifically, aphotoelectric conversion region that corresponds to the pupil regions707C, 707F, 707I, and 707L is formed to serve as a first region.Additionally, a photoelectric conversion region corresponding to thepupil regions 707B, 707E, 707H, and 707K, serving as an annularphotoelectric conversion region provided outside the first region, isalso formed. Note that a signal obtained from an annular photoelectricconversion region provided outside the first region is referred to as afourth signal. Additionally, the pupil region 6073 is a pupil regiondetermined by the open aperture of the shooting lens unit 500 asdescribed above.

FIG. 12 illustrates an example of a cross-sectional configuration in thex direction passing through the center of the pixel corresponding toFIG. 11A, and relates to three pixels 200, 300, and 400 arranged asshown in FIG. 8. The lower electrode of each pixel is formed by a lowerelectrode divided into 12 portions, and each lower electrode has asimilarity shape corresponding to the pattern of the pupil region shownin FIG. 11A. Additionally, each pixel also has 12 signal readout units,209A to 209K, 309A to 309K, 409A to 409K . . . connected to each lowerelectrode. The signal-readout from the 12 signal readout units may beconfigured to read out signals individually or the signals may be readout after addition processing that may be performed on the signals inthe image pickup device.

Images for image recording or display having three different F-numberscan be simultaneously acquired by one image pickup (shooting) by usingthe image pickup device having a lower electrode with a similar shapecorresponding to the pupil regions as shown in FIG. 11A. For example, itis assumed that the open F-number of the shooting lens unit is F 2.0.Here, it is possible to acquire images having F 2.8 (first-stepaperture) and F 4.0 (second-step aperture) at one time. A method forreading out signals from the pupil region at that time will be describedbelow.

Note that the signals from the lower electrodes corresponding to thepupil regions 707A to 707L are defined as 707AS to 707LS. In the case ofF 2.0, addition is performed, for example, as707AS+707BS+707CS+707DS+707ES+707FS+707GS+707HS+707IS+707JS+707KS+707LS.Specifically, the first to fourth signals are used to form a first imagesignal. In the case of F 2.8, addition is performed, for example, as707BS+707CS+707ES+707FS+707HS+707IS+707KS+707LS. Specifically, a thirdimage signal is formed by using the first signal and the fourth signal,instead of using the second and third signals. Such a mode that formsthe third image signal is referred to as a “third mode”. In the presentembodiment, the image processing means can select the third mode inaddition to the first and second modes.

In the case of F 4.0, the image processing means performs addition, forexample, as 707CS+707FS+707IS+707LS, to form the second image signal. Insignal-readout from each pixel in FIG. 12, images having F-numbers, F2.0, F 2.8, and F 4.0 on the image pickup device 101 can be obtained bygenerating the above-mentioned addition signals by using an imagegenerating means.

Additionally, in focus detection, as described with reference to FIG. 6,parallax images having centroid position G_A to G_L are generatedcorresponding to each pupil region by using the focus detection means.At this time, various methods are conceivable for obtaining the imageshift amount p of the object image between the parallax images. Forexample, the left half pupil region in FIG. 11A is obtained by thecalculation 707 LEFT=707AS+707BS+707CS+707JS+707KS+707LS. In this case,the centroid position becomes G_LEFT=(G_A+G_B+G_C+G_J+G_K+G_L)/6.Furthermore, the right half pupil region is obtained by the calculation707 RIGHT=707 DS+707ES+707FS+707GS+707HS+707IS. In this case, thecentroid position becomes G_LEFT=(G_D+G_E+G_F+G_H+G_I+G_J)/6. The focusdetection is performed by calculating the image shift amount p based onthe G_LEFT and the G_RIGHT and obtaining the defocusing amount d.

Similarly, G_UP=(707AS+707BS+707CS+707DS+707ES+707FS)/6 is obtained byusing the vertical parallax. Additionally,G_DOWN=(707GS+707HS+707IS+707JS+707KS+707LS)/6 is obtained. Focusdetection in the vertical direction may be performed by calculating theimage shift amount p based on these differences and obtaining thedefocusing amount d.

In addition, for example, focus detection can be performed only by usingparallaxes (for example, G_A and G_D.) by using the signals from asingle pupil region for each or focus detection may be performed byusing any other combination. In the image pickup device described inEmbodiment 2, as shown in FIG. 11A, since pupil division in the verticaldirection and pupil division in the horizontal direction are mixed, thecombination of parallax images used during focus detection can bechanged in various ways. Additionally, it may be possible to change thecombination of the pupil regions for performing focus detectiondepending on, for example, a two-dimensional pixel position on the imagepickup plane, an angular direction of the object, F-number duringshooting, and a lens type to be mounted.

Next, the relation between the pupil regions 707A to 707L and the signalreadout method in FIG. 11A will be described. As shown in FIG. 11A, ifthe number of the pupil divisions is increased and the image pickupdevice signals corresponding to the respective pupil regions areindependently read out and stored in the camera, this is convenient whenimages having different F-numbers are generated in the post-processing(for example, a personal computer). However, the number of channels readout from the signal readout unit of FIG. 12 also increases accordingly.In the case of performing the signal readout, 12 channels are requiredfor signal readout when the number of pupil division is 12 as shown inFIG. 11A.

In contrast, in order to reduce the number of readout channels and theamount of information to be stored in the storage medium 106 inperforming processing in the camera, the following measures may be used.FIG. 11C and FIG. 11D illustrate seven kinds of pupil region FA to pupilregion FC and pupil region SA to pupil region SD. As shown in FIG. 11Cand FIG. 11D, seven kinds of pupil regions may be set, and the imagepickup signals corresponding to these pupil regions may be read out. Atthat time, electric charges may be added in the circuit of thesemiconductor substrate 208 of the image pickup device and read out.

That is, for the pupil region FA having a circular shape located at thecenter in FIG. 11C, 707CS+707FS+707LS+707IS is calculated. For the pupilregion FB having an annular shape in FIG. 11C,707BS+707ES+707HS+707KS+FAS is calculated. For the pupil region FC thatis a dot region in FIG. 11C, 707AS+707DS+707GS+707JS+FBS is calculated.For the pupil region SA having a rectangular shape located at the upperleft of FIG. 11D, 707AS+707BS+707CS is calculated. For the pupil regionSB having a rectangular shape located at the upper right of FIG. 11D,707DS+707ES+707FS is calculated. For the pupil region SC having arectangular shape located at the lower right of FIG. 11D,707GS+707HS+707IS is calculated. For the pupil region SD having arectangular shape located at the lower left of FIG. 11D,707JS+707KS+707LS is calculated.

In the above description, the pupil region FA is, for example, a pupilregion corresponding to a shooting image having F 4.0. Since the pupilregion FA+FB is a pupil region corresponding to a shooting image havingF 2.8 and the pupil region FA+FB+FC represents the overall pupil region,the pupil region corresponds to the shooting image having F 2.0 (open).That is, in order to obtain the shooting images having a plurality ofdifferent F-numbers, pixel signals (a plurality of addition signals)corresponding to the pupil regions FA, FB, and FC are read out.

Next, the pupil region SA to the pupil region SD are used as pupilregions for focus detection. During phase difference focus detection,focus detection is performed by using, for example, parallax images ofthe pupil region SA+SB and the pupil region SC+SD, and a parallax isgenerated in the vertical direction, and thereby the phase differencefocus detection described in Embodiment 1 can be performed.

Similarly, focus detection is performed by using parallax images of thepupil region SA+SD and the pupil region SB+SC and a parallax isgenerated in the horizontal direction, and thereby the phase differencefocus detection described in Embodiment 1 can be performed. Phasedifference focus detection can be performed by reading out pixel signals(a plurality of addition signals) respectively corresponding to thepupil regions SA, SB, SC, and SD. As described above, signals can beread out from the image pickup device with a number of channels that issmaller than the divided number of the pupil region (the divided numberof the lower electrode) by using a plurality of addition signals, and asa result, it is possible to generate a captured image and perform phasedifference focus detection.

It may be possible to obtain focus detection capabilities in thevertical direction and the horizontal direction simultaneously bychanging and mixing various readout methods for focus detection asdescribed above in each pixel unit or in units of a plurality of pixelsarranged in the XY plane of the image pickup device shown in FIG. 8.Additionally, although, in FIG. 11A, the pupil region is divided alongthe X axis and the Y axis, the pupil region may be divided along an axisrotated by 45 degrees in the oblique direction of the X axis and the Yaxis as shown in FIG. 13A. Additionally, as shown in FIGS. 13B and 13C,a shape in which the number of the divided portions for a part of thepupil region is small may be possible. The number of patterns of thelower electrode can be reduced by reducing the number of pupildivisions, and the number of readout channels from the image pickupdevice can be reduced. Therefore, it is possible to reduce, for example,the processing speed and the processing load.

Note that, in the above embodiments, although an example of forming aplurality of divided photoelectric conversion regions by the combinationof the divided lower electrodes, the organic photoelectric conversionfilm 204, and the upper electrode has been described, the presentinvention is not limited to such a configuration. For example, the upperelectrode may be divided into a plurality of portions, and a pluralityof photoelectric conversion regions may be formed by the upperelectrodes. In that case, the shape of the divided photoelectricconversion regions is determined depending on the shape of the dividedupper electrodes. Additionally, for example, in an image pickup devicein which an inorganic material is used for the photoelectric conversionunit, a plurality of photoelectric conversion regions may be formed byarranging a plurality of photodiodes beneath the micro lens in eachpixel and forming a light receiving surface of each photodiode so as tohave a shape similar to that in the present embodiment.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2019-020052, filed on Feb. 6, 2019, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: animage pickup device configured to have a plurality of pixels arrangedalong an image pickup plane, the pixel having a micro lens forcondensing light from outside into the pixel and a photoelectricconversion region provided beneath the micro lens and for generating aphotoelectric conversion signal, the photoelectric conversion regionincluding an upper electrode and a lower electrode sandwiching aphotoelectric conversion film, the upper electrode or the lowerelectrode being divided into a plurality of portions, and forming atleast a first region, a second region, and a third region that aredivided and arranged in a plane parallel to the image pickup plane bythe divided upper electrodes or the lower electrodes, the first regionhaving a shape for forming a circle, the second region and the thirdregion being arranged outside the first region; and a readout unitconfigured to read out the photoelectric conversion signals obtained byone image pickup operation to serve as respectively first, second, andthird signals from the first, second, and third regions.
 2. The imageprocessing apparatus according to claim 1, further comprising an imagesignal processing unit configured to have a first mode that forms afirst image signal by adding the first, second, and third signals fromamong the first, second, and third signals that have been read out bythe readout unit, and a second mode that forms a second image signal byusing the first signal without using the second and third signals. 3.The image processing apparatus according to claim 1, further comprisinga focus detection unit configured to generate a focus detection signalby using at least the second and third signals from among the first,second, and third signals that have been read out by the readout unit.4. The image processing apparatus according to claim 1, furthercomprising a storage unit configured to store each of the first, second,and third signals that have been read out by the readout unit.
 5. Theimage processing apparatus according to claim 1, wherein the firstregion is further divided into a plurality of regions.
 6. The imageprocessing apparatus according to claim 1, wherein the photoelectricconversion region has an annular region provided outside the firstregion.
 7. The image processing apparatus according to claim 6, whereinthe readout unit reads out a fourth signal from the annular region. 8.The image processing apparatus according to claim 5, wherein the annularregion is divided into a plurality of regions.
 9. The image processingapparatus according to claim 7, wherein the image processing unit canselect a third mode that forms a third image signal by using the firstsignal and the fourth signal instead of using the second and thirdsignals.
 10. The image processing apparatus according to claim 1,wherein the photoelectric conversion region is divided into a pluralityof regions, and the readout unit adds and reads out a part of signals ofthe regions.
 11. The image processing apparatus according to claim 1,wherein the photoelectric conversion region is divided into a pluralityof regions, and the readout unit has a storage unit for reading out andstoring signals from the regions.
 12. The image processing apparatusaccording to claim 11, further comprising a unit configured to change acombination of signals from the regions stored in the storage unit andperform addition.
 13. The image processing apparatus according to claim1, wherein the photoelectric conversion film includes an organicphotoelectric conversion film.
 14. The image processing apparatusaccording to claim 2, wherein the image pickup device and the readoutunit are built in an image pickup apparatus, and the image signalprocessing unit is built in an external apparatus separate from theimage pickup apparatus.
 15. An image processing apparatus comprising: animage pickup device having a plurality of pixels arranged along an imagepickup plane, the pixel having a micro lens for condensing light fromoutside into the pixel and a photoelectric conversion region providedbeneath the micro lens and for generating a photoelectric conversionsignal, the photoelectric conversion region including an upper electrodeand a lower electrode sandwiching a photoelectric conversion film, theupper electrode or the lower electrode being divided into a plurality ofportions, and forming at least a first region, a second region, and athird region that are divided and arranged in a plane parallel to theimage pickup plane by the divided upper electrodes or the lowerelectrodes, the first region having a shape for forming a circle, thesecond region and the third region being arranged outside the firstregion; a readout unit configured to read out photoelectric conversionsignals obtained by one image pickup operation to serve as respectivelyfirst, second, and third signals from the first, second, and thirdregions; an image signal processing unit configured to have a first modethat forms a first image signal by adding the first, second, and thirdsignals from among the first, second, and third signals that have beenread out by the readout unit, and a second mode that forms a secondimage signal by using the first signal instead of using the second andthird signals; and a focus detection unit configured to generate a focusdetection signal by using at least the second and third signals.